1 METHOD FOR MAKING A COLORFUL 3D MODEL 

2 BACKGROUND OF THE INVENTION 

3 1. Field of the Invention 

4 The present invention relates to a method for constructing a three 

5 dimensional model, and more particularly to a method using the transformation 

6 of a generic model with regular mesh structure embedded therein to have the 

7 regularly transformed mesh structure. Further, the method is able to 

8 automatically compensate color difference between two adjacent transformed 

9 mesh to reach a highly realistic surface effect. 

10 2. Description of Related Art 

1 1 Nowadays, a computer generated three dimensional (3D) model has 

12 been widely used in different fields, i.e. from the characters in video games to 

13 special visual effect in the movie industry or from the commercial development 

14 of multi-media to the special requirements in the medical industry. As a 

15 consequence, the construction and operation of 3D data or 3D model have 

16 become crucial lessons in the field of making a 3D model. 

17 The conventional way of making a 3D model starts from the drafting of 

18 the animation engineer by using the modeling software. Normally, it took a long 

19 time to train a qualified animation engineer. After the animation engineer is 

20 qualified, the engineer still has to use creativity to add in "personal touch" in the 

21 modeling process and also the coloring to the finished model to make the model 

22 as much perfect as possible. This art creation process takes a long time to finalize 

23 the entire process. Also, the "personal touch" sometimes may become the 
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1 greatest failure in the entire creation process. 

2 In contrast to the conventional model creation method, using 

3 measurement devices to construct a 3D model belongs to the category of reverse 

4 engineering. The shape and color information can be retrieved by using delicate 

5 devices with the 0.01cm or higher accuracy. The measured data of the shape and 

6 color of an object usually is presented by a triangular mesh or curved surface to 

7 show the geometry information, which is shown in Fig. 1A. A two dimensional 

8 image is shown in Fig. IB to show the color information. The interrelationship 

9 between the color information and the geometry information is shown by texture 

10 mapping. The mapping is often referred as texture coordinate. In order to have a 

1 1 complete model, different measurement angle to the object is required. Then the 

12 measured data is adjusted and integrated into the same spatial coordinate system, 

13 which is shown in Fig. 1C. Thereafter, the data is integrated into a complete 3D 

14 model as shown in Fig. ID. 

15 The model created by reverse engineering process has high accuracy in 

16 relation to the object. The difference is hardly to be recognized by naked eyes. 

17 Besides, there is no special training program required for the operator. The 

18 operator only needs to be familiar with the equipment. However, the data 

19 obtained from the measurement instrument usually is enormous and lack of 

20 regularity, which results in that the data can only be used in the production of a 

21 specific object. Besides, the large quantity of data hinders the post-process, e.g. 

22 data transmission or data reproduction. Furthermore, as an influence from the 

23 light, the data from different measuring angle has obvious color difference. 
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1 Therefore, a complete method to practically use the original measured data is 

2 required to solve the previously described problems. 

3 To mend the problems, some recommends to construct a 3D model by 

4 using special tools. Still the time spent for manually constructing a 3D model 

5 does not meet the cost-effectiveness requirement. Due to the fast growth of 

6 reverse engineering, highly precise measurement instrument is applied to 

7 retrieve object's 3D data to recreate a vivid model out of the object measured. 

8 U.S. Pat. No. 6,512,518 (the '518 patent) discusses a method of using a 

9 laser scanning device to retrieve object 3D data and then the obtained data is 

10 transformed into meshed data. A method for integrating the meshed data is also 

1 1 provided. The c 5 18 patent is able to quickly and accurately measure the spatial 

12 position of an object so that a highly accurate model is produced. However, the 

13 spatial position is represented by a densed dot group data, which is large and 

14 irregular. Consequently, the re-use of the measured data is highly unlikely. U.S. 

1 5 Pat. No. 6,356,272 ('272 patent) applies shape from silhouette principle to utilize 

16 fixed camera system to take large amount of pictures to create a 3D model from 

17 the continuous images and establish the mapping relationship between the 

18 images and the mesh. The pictures taken by the '272 patent are continuous from 

19 the sides of the object. The best mapping relationship is chosen from the angle 

20 between the normal of a triangle and the image. The top and bottom of an object 

21 or an object with a complex appearance may have data distortion when mapping 

22 occurred. 

23 To overcome the shortcomings, the present invention tends to provide an 
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1 improved method to make a vivid and colorful model to mitigate the 

2 aforementioned problems. 

3 SUMMARY OF THE INVENTION 

4 The primary objective of the present invention is to provide an improved 



5 method which is able to integrate the retrieved data into a complete colorful 

6 information so as to establish a vivid 3D model Besides, the retrieved data from 

7 an object is mapped to a generic model having regular data embedded therein. 

8 After mapping, the data from the generic model is forced to transform into usable, 

9 regular geometry information for the model. 



10 Another objective of the present invention is to provide a color mending 

1 1 method to compensate color difference between adjacent data such that the 

12 surface color on the model is smooth and continuous. 

13 Other objects, advantages and novel features of the invention will 

14 become more apparent from the following detailed description when taken in 

15 conjunction with the accompanying drawings. 

16 BRIEF DESCRIPTION OF THE DRAWINGS 

17 Fig. 1A is schematic view to show the geometry information of a picture 

18 by using the triangular mesh or a curved surface; 

19 Fig. IB is a schematic view showing the color information of a two 

20 dimensional image; 

21 Fig. 1C is a schematic view showing the integration of all the measured 

22 data into the same spatial coordinate; 

23 Fig. ID is a schematic view showing that all the measured data is 
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1 integrated into a complete three dimensional model; 

2 Fig. 2 is a flow chart showing the production of the 3D model; 

3 Fig. 3 A is a schematic view showing the original measured mesh by the 

4 measuring instrument; 

5 Fig. 3B is a schematic view of mesh of a new model with precise and 

6 regular data; 

7 Fig. 3C is a schematic view showing color difference between adjacent 

8 mesh; 

9 Fig. 4 is a flow chart of reconstructing regular mesh model; 

10 Fig. 5 is a schematic view showing the original mesh and the 

1 1 corresponding color information; 

12 Fig. 6 is a schematic view of the selected generic model; 

13 Fig. 7 is a schematic view of the transformed generic model; 

14 Fig. 8 A is a schematic view showing the original measured data; 

15 Fig. 8B is a schematic view of the reconstructed model by using the 

16 generic model of Fig. 6; 

17 Fig. 9 is a schematic view showing that the texture image data is 

18 extracted from the original measured data; 

19 Fig. 10 is a flow chart of abstracting color map information; 

20 Fig. 11 A is a schematic view showing the spatial interrelationship 

2 1 between the texture image data of the original measured data and the generic 

22 model ; 

23 Fig. 1 IB is a schematic view showing that the texture image data is 
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1 reattach to the generic model to complete the color abstracting process; 

2 Fig. 12 is a flow chart showing the harmonization of color between two 

3 measured mesh; 

4 Fig. 13 is a schematic view of the overlapping relationship and the 

5 arrangement sequence of the measured data; 

6 Fig. 14 is a schematic view showing that the overlapped portion of two 

7 adjacent mesh model and the overlapped portion corresponds to respective 

8 texture image; 

9 Figs. 15A and 15B are comparison between the mesh models before and 

10 after adjustment of color average; 

1 1 Fig. 16 is a flow chart showing the pixel blending; and 

12 Fig. 17 A, 17B and 17C are schematic views showing the advanced 

13 comparison result from Figs. 15 A and 15B. 

14 DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 

1 5 The present invention relates to a method of processing three 



16 dimensional (3D) data to integrate the measured 3D data from the object to be 

17 reproduced into a complete 3D color model In the geometry information aspect, 

18 the method applies a generic model to combine measured data from different 

19 angles of the object to become a mesh model with regular data embedded 

20 therein. 

21 In the color information aspect, the method applies the comparison 

22 between the spatial corresponding relationship of the newly produced regular 

23 data of the mesh model and the original measured data to reattach the texture 
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1 image data of the measured data to the model. The color difference between 

2 adjacent images is then adjusted so that by. means of interactive measurement, 

3 the operator is able to easily construct a 3D model with high accuracy and 

4 applicability. 

5 The method uses a generic model to circumstantially integrate the 

6 original measured data into a complete model. The word "generic" means that 

7 the model is applicable to all sorts of objects with the similar appearances such 

8 that severe distortion may be avoided. For example, to construct a human head 

9 model, a generic model with facial characteristics i.e. a pair of eyes, a nose, a 

10 mouth and a pair of ears may be applied. To construct an animal such as a cow, a 

1 1 horse or even a sheep, a generic model with four legs may be applied. 

12 The original and vast mesh of the object is not dealt in the present 

13 invention but to adopt the pre-designed generic model with regular mesh 

14 structure to map with the original measured data such that a rough model with 

15 the same appearance as that of the object being measured is built. If there is any 

16 data breakage such as hair or other parts of the object that are not easy to be 

17 measured, the breakage may be mended automatically by applying the mesh 

18 structural relationship between adjacent data in the mapping process. The 

19 corresponding relationship of the texture images is automatically established by 

20 using the spatial relationship between the generic model and the measured data 

21 without the involvement of special positioning equipment or any manual 

22 operation. 

23 The method of the present invention mainly is divided into four major 
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1 parts: 

2 reconstructing regular mesh model; 

3 abstracting color; 

4 harmonization of color arrangement; and 

5 pixel blending between overlapped images. 

6 With reference to Fig. 2, the first step of the present invention is to 



7 reconstruct regular mesh model. The data measured by the three dimensional 

8 measuring device is dense, as shown in Fig. 3 A, to reduce the error by replacing 

9 the curved surface model with the mesh model. This is particular true for those 

10 objects with complicated shapes or fine minute characteristics. However, the 

1 1 more accurate the measured data is, the larger quantity of the triangular mesh it 

12 becomes. Therefore, direct implication of the original measured data may lead to 

13 a consequence that the quantity of the meshes is large and is not applicable. 

14 Therefore, using a generic model with regular mesh structure embedded 

1 5 therein to map with the original measured data to generate a new model. The new 

1 6 model has a regular mesh structure inherited from the generic model; meanwhile, 

17 it is transformed to a shape similar to the original measured data, as shown in Fig. 

18 3B. Further, due to the overlapping relationship in the spatial positions between 

19 the original measured data and the data of the new model, the texture images 

20 from the original measured data are projected to the new model so as to establish 



21 the corresponding relationship between the new model and the texture images. 

22 When the second step is finished, the construction of a complete model 

23 with regular mesh structure and multiple color texture images is completed. 
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1 However, due to the color difference between the texture images takes from 

2 different viewing angles, as shown in Fig. 3C, the overlapping areas of images is 

3 used to adjust the color difference to allow the brightness of all the images 

4 becomes the same. Pixel blending is processed in the image overlapping area so 

5 that a 3D model, as shown in Fig. 3D, with concise mesh structure and smooth 

6 surface color is generated. 

7 With reference to Fig. 4, the original measured data (100) is a group of 

8 mesh models obtained by 3D measuring device. Each model is composed of 

9 mesh data (110) and texture image data (120), obtained by measurement in 

10 different angles to an object to be reproduced. All of the models are then 

1 1 transformed into the same coordinate system.. In step (SI 02), according to the 

12 shape of the object, a generic model (200) with the similar appearance to the 

13 object is selected. In step (S 104), the generic model is roughly overlapped with 

14 the original measured data in space. In step (S106), the dimension of the generic 

15 model is adjusted to correspond to the dimension of the original measured data. 

16 In last step (S108), the generic model is projected to the original model. 

17 Consequently, the data of the generic model is deformed and thus the generic 

18 model has an appearance similar to that of the original measured data (100). 

19 Even so, the data of the generic model still has regular mesh structure 

20 characteristic. 

21 Figure 6 shows a generic model (200) ready for use in the present 

22 invention. Figure 7 shows the appearance change of the deformed generic model 

23 (210). Figure 8 A and Figure 8B show the differences in the quantity of mesh and 
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1 mesh distribution between the original measured data (100) (Fig. 8 A) and the 

2 deformed generic model (210). 

3 Color abstracting is to separate texture image data (120) from the 

4 original measured data (100). Then the texture image (120) is re-mapped to the 

5 deformed generic model (210), as shown in Fig. 9. As a matter of fact, to 

6 establish the corresponding relationship between the deformed generic model 

7 (210) and the texture image (120), the texture coordinate and the corresponding 

8 texture image of each mesh point of the deformed generic model (2 10) is 

9 required. Because each mesh point of the deformed generic model (210) is 

10 projected to the original measured data (100), the triangle contains the projected 

1 1 mesh point is used to calculate the texture coordinate and the texture image 

12 corresponding to the triangle of the deformed generic model mesh point is used 

13 as the corresponding texture image. 

14 With reference to Fig. 10, step (S202) is to choose the corresponding 

15 triangle of mesh point of the deformed genetic model (210). Step (S204) is to 

16 calculate the texture coordinate of the chosen triangle of the mesh point of the 

17 deformed generic model (210) to correspond to the texture image. In step (S206), 

18 check continuity of the triangles chosen to see if the chosen triangles are within 

19 the same texture image. If the coordinates of the chosen triangles are not 

20 continuous, not within the same texture image, other triangles should be selected 

21 to calculate the corresponding coordinates, which is shown in step (S208). Then 

22 to complete the calculation of the coordinates of all the triangles of the deformed 

23 generic model (210), in step (S210), a process is required to check if all the 
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1 triangles of the deformed generic model (210) are calculated. 

2 With reference. to Figs. 11A and 11B, after color abstraction, the generic 

3 model (220) is a three dimensional colorful model and contains multiple texture 

4 images. However, because the texture images are taken from different angles, 

5 color differences between the texture images may occur. In order to harmonize 

6 the surface color of the generic model (220), the overlapped characteristic 

7 between the texture images is used. 

8 With reference to Fig. 12, step (S302) is to seek the overlapped area Oy 

9 between measured data (100). That is, if the measured data (100) is M={M!, M 2 , 

10 M 3 ,. . .M n }, n three dimensional mesh, 0 {j stands for the overlapped area between 

1 1 any two adjacent measured data M ; , Mj. In step (S304), the magnitude of O y is 

12 determined. Then in step (S306), the sequence of M is determined. Therefore, if 

13 M x is the first layer M L1 , all the mesh model related to and overlapped with M x is 

14 M L2 . Thus all the mesh model related to M L2 is M L3 and so on. Each mesh model 

1 5 in each layer is arranged in a descending manner according to their magnitudes. 

16 Thus a new three dimensional mesh model group M' ={M' ,,M' 2 >„.,M' n } is 

17 obtained. 

18 Figure 13 shows the overlapping relationship and the layer sequence of the 

19 measured data. Figure 14 shows the overlapped area between two adjacent mesh model 

20 and the overlapped areas respectively correspond to their own texture images. 

2 1 In step (S308), according to M' sequence, the color adjustment value Aj of the 

22 texture image of the mesh model is calculated in relation to the intensity average of the 

23 overlapped area of each mesh model, which is: 
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1 Intensity average value of the overlapped area of M'i is: I AV G,i = 1 ,2,3, . . . ,n 

2 Color adjustment value of NT { A x = 1 

3 Then the color adjustment value of M\ influenced by M'j is A { ^=A X X 

4 (Iavg,i/Iavg,i) 

5 Then if all the mesh models that are overlapped with M' { are taken into 

6 consideration, color adjustment value of M' i would be 

7 AKAi.1 X W u +. . . >A bM X W i , i . 1 )/(W U +. . 

8 where W s is the mesh influenced weight value. 

9 Figure 15 shows the comparison between and after color average 

10 adjustment of a group of mesh models, wherein Fig. 15 A is before the color 

1 1 adjustment and Fig. 15B is after color adjustment. 

12 Pixel blending is processed to the images in the overlapped areas to 

13 harmonize the color of two adjacent images. 

14 With reference to Fig. 16, in step (S402), it is to seek all the overlapped 

15 triangles and the texture images covered by the triangles. To triangle T, if the 

16 corresponding texture images are I t ,iJt,2- • Jt,™ the m texture images is 

17 overlapped in all the parts of T t ,i,T u „„Ti,m in relation to triangle T. Therefore, 

18 pixel blending is processed to these overlapped mapped areas. 

19 In step (S404), to each triangle T in the overlapped area, calculation of 

20 the distances D of the vertices of the triangle T to the nearest boundary vertex is 

2 1 required. Because the triangle T has m corresponding mesh models, the distances 

22 Dl, D2. . .,Dm are obtained by calculating each vertex of the triangles. In step 

23 (S406), each triangle in the overlapped area is used as an unit so that pixel 

12 



blending weight average is processed to the texture image corresponding area 
covered by the unit. To the vertex Vi(i= 1,2,3) of each triangle, the weight of the 
pixel blending is D u ,D i>2 ,....D i>fn . The pixel color of the covered images is 
C u ,C i 2 , . .C im . The color after pixel blending is Q AVG . To every sampling point 
within the triangle, the pixel blending weight is calculated by applying the 
barycentric coordinate principle. Then the color after pixel blending is processed 
by using the same principle. 

Q, A vg=(C u X D u +Q, 2 X D i>2 . . .+(Q, m X D itn ) 

Fig. 17 is an advanced comparison to Fig. 15, wherein Fig. 17C is the 
result after pixel blending to the overlapped area of Fig. 17B. 

In reference to the following tables, it is appreciated to learn the 
advantages of the present invention. 



^\Method 


Conventional 
method 


.S. Pat. 
6,512,518 


U.S. Pat. 
6,356,272 


Present 
invention 


Owner 




Cyra 


Sanyo Electric 


ITRI 


Treatment 


Manual 


Interactive 


interactive 


Interactive 


Constructing 
time 


Longest 


Long 


Short 


Short 


Mesh structure 


Regular 


Irregular 


Irregular 


Regular 


Texture mapping 


Manual 




Auto-mapping 


Auto-mapping 


Color evenness 


Excellent 




bad 


Excellent 
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Appearance 
similarity 


Fair 


Good 


Good 


Excellent 


Reusability 


Excellent 


Bad 


Bad 


Excellent 


Others 








Auto-repair to 
data 

discontinuity 
(such as hair) 



1 

2 It is to be understood, however, that even though numerous 

3 characteristics and advantages of the present invention have been set forth in the 

4 foregoing description, together with details of the structure and function of the 

5 invention, the disclosure is illustrative only, and changes may be made in detail, 

6 especially in matters of shape, size, and arrangement of parts within the 

7 principles of the invention to the full extent indicated by the broad general 

8 meaning of the terms in which the appended claims are expressed. 
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